Behavior coordination for a mobile robot using modular reinforcement learning

Coordination of multiple behaviors independently obtained by a reinforcement learning method is one of the issues in order for the method to be scaled to larger and more complex robot learning tasks. Direct combination of all the state spaces for individual modules (subtasks) needs enormous learning time, and it causes hidden states. This paper presents a method of modular learning which coordinates multiple behaviors taking account of a trade-o between learning time and performance. First, in order to reduce the learning time the whole state space is classi ed into two categories based on the action values separately obtained by Q learning: the area where one of the learned behaviors is directly applicable (no more learning area), and the area where learning is necessary due to the competition of multiple behaviors (re-learning area). Second, hidden states are detected by model tting to the learned action values based on the information criterion. Finally, the initial action values in the relearning area are adjusted so that they can be consistent with the values in the no more learning area. The method is applied to one to one soccer playing robots. Computer simulation and real robot experiments are given to show the validity of the proposed method.